Method of processing and recognizing hand-written characters

ABSTRACT

The present disclosure relates to a method and system of processing original handwriting input, the system and method being capable of recognize a plurality of strokes provided on the input recognition interface, the method including: determining a stroke box around each stroke; determining overlap between the stroke boxes; correlating overlapping stroke boxes to one or more characters; providing a character box around each of the one or more characters; determining overlap between character boxes; correlating overlapping character boxes to one or more words; providing a word box around each of the one or more words; provide a word margin around each of the one or more word boxes; determining overlap between each word box to determine a line; wherein each of the characters, words, or lines can be individually selected and rearranged, the system automatically adjusting spacing or placement of surrounding elements to allow for the rearrangement.

PRIORITY

This application claims priority to U.S. provisional application No.62/345,928, which was filed on Jun. 6, 2016, and U.S. Non-provisionalapplication Ser. No. 15/615,215, which was filed on Jun. 6, 2017 arehereby incorporated by reference in its entirety.

COPYRIGHT STATEMENT

A portion of the disclosure of this patent document contains materialwhich is subject to (copyright or mask work) protection. The (copyrightor mask work) owner has no objection to the facsimile reproduction byanyone of the patent document or the patent disclosure, as it appears inthe Patent and Trademark Office patent file or records, but otherwisereserves all (copyright or mask work) rights whatsoever.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The disclosure relates to word processing, particularly in regard torecognition of handwritten notes in electronic media through the gesturetracking and analysis of handwritten characters or strokes provided by afinger or stylus.

2. Description of the Prior Art

Present electronic writing systems typically seek to translate orotherwise convert optically recognized characters or strokes of a fingeror stylus into an electronic font format, for example Arial, Calibri, orTimes New Roman. In additional systems, pre-programmed stroke sequencesare configured to correspond to particular letters of the alphabet. Thepioneers of such systems including the Palm™ series personal organizers.However, the personal touch is somewhat lost because these systemsalmost always remove the hand-written strokes and replace them with fonttype characters. One of the goals of the present disclosure is toprovide a system which retains the actual strokes and recognizes thehand-written words while allowing a certain degree of manipulation yetretaining the images of the original input strokes and thus the look ofnatural handwriting.

SUMMARY

Contemplated herein is a handwriting recognition and processing systemwhich can include a non-transitory computer-readable medium; an inputrecognition interface; and processing circuitry operably connected tothe non-transitory computer-readable medium and the input recognitioninterface. In various embodiments as contemplated herein the processingcircuitry being configured to perform the following tasks: recognize aplurality of strokes provided on the input recognition interface;determine a stroke box around each stroke; provide a stroke marginaround each of the stroke boxes; determine overlap between the strokemargins for each stroke box; correlate overlapping stroke margins to oneor more characters; provide a character box around each of the one ormore characters; and provide a character margin around each of the oneor more character boxes.

In yet additional embodiments, the processing circuitry can beconfigured to determine overlap between the character margins for eachcharacter box; and correlate overlapping character margins to one ormore words.

In yet additional embodiments, the processing circuitry can also beconfigured to differentiate between a plurality of input methodsincluding at least a stylus input and a touch input. In some suchembodiments, input from the stylus can be associated with a positivestroke, and touch input can be alternatively associated with analternative command. In such a case, at least one alternative commandcan be associated to an undo command regarding at least one previousstroke.

In yet additional embodiments, the input recognition interface can beconfigured to receive a positive inference stroke connecting a pluralityof separated strokes from the stylus, and a touch input received todelete the positive inference stroke, wherein the processing circuitrycan then retain inference information between the strokes so as torecognize the plurality of separated strokes as an associated character,word, or line.

In addition to strokes and stroke boxes, additional classes of charactergroups can be created, for example for words and lines. In some suchinstances, the processing circuitry can then be configured to provide aword box around each of the one or more words; provide a word marginaround each of the one or more word boxes; determine overlap between theword margins for each word box; correlate overlapping word margins toone or more lines; and determine a line box correlating to the verticaland horizontal bounds of each line.

Additionally, in some embodiments where the processing circuitryrecognizes a distinct line of characters or words, the processingcircuitry can then determine a reference line or central line for eachline box; and determine an offset of each word with respect to thereference or central line.

In yet additional embodiments the processing circuitry can be furtherconfigured to perform the following tasks: calculating an average wordspacing; allow selection of an individual word; and allow forrepositioning of the individual word within an associated reference orcentral line; wherein one or more surrounding words are configured to beautomatically repositioned along the reference or central line to allowfor repositioning of the individual word in accordance with the averageword spacing on the reference or central line having a proper associatedoffset with respect to the reference or central line.

Similarly, the processing circuitry can be further configured to performthe following tasks: calculate an average character spacing; determinean offset of each character with respect to an associated reference orcentral line; allow selection of an individual character within anassociated word; and allow for repositioning of the individual characterwithin the associated word along the associated reference or centralline; and wherein one or more surrounding characters are configured tobe automatically repositioned along the reference or central line toallow for repositioning of the individual character in accordance withthe average character spacing on the associated reference or centralline having a proper associated offset with respect to the reference orcentral line.

In yet further embodiments, the processing circuitry can be furtherconfigured to perform the following tasks: calculate an average wordspacing; calculate an average character spacing; receiving user inputregarding selection of an individual word or character; and receive auser input correlating to a deletion command of the word or characterwithin an associated word or line; wherein one or more surrounding wordsor characters are repositioned along the reference or central line toallow for repositioning of the individual word or character inaccordance with the average word or character spacing.

Also contemplated herein is a method of recognizing and processinghand-written strokes and characters, the method including the steps of:receiving a plurality of strokes provided by using a writing utensil onan input recognition interface; recognizing the plurality of strokesprovided on the input recognition interface utilizing processingcircuitry; determining a stroke box around each stroke; providing astroke margin around each of the stroke boxes; determining an overlapbetween the stroke margins for each stroke box; correlating anyoverlapping stroke margins to one or more characters; providing acharacter box around each of the one or more characters; providing acharacter margin around each of the one or more character boxes;determining overlap between the character margins for each characterbox; correlating overlapping character margins to one or more words; andstoring stroke and character information on a non-transitorycomputer-readable medium.

The method can also include a step of: differentiating between aplurality of input methods including at least a tactile touch input andinput received by the writing utensil, wherein input from the writingutensil is associated with a positive stroke, and touch input isassociated with an alternative command.

In some additional embodiments, the method can include situations inwhich at least one alternative command is an undo command regarding atleast one previous stroke.

The method can also include the steps of: receiving, from the writingutensil, a positive inference stroke connecting a plurality of separatedstrokes; and receiving a touch input so as to delete the positiveinference stroke, wherein the processing circuitry causes thenon-transitory computer-readable media to retain inference informationbetween the strokes so as to recognize the plurality of separatedstrokes as an associated character, word, or line.

The method can also include the steps of: providing a word box aroundeach of the one or more words; providing a word margin around each ofthe one or more word boxes; determining an overlap between the wordmargins for each word box; correlating overlapping word margins to oneor more lines; determining a line box correlating to the vertical andhorizontal bounds of each line; determining a reference or central linefor each line box; and determining an offset of each word with respectto the reference or central line.

Similarly, the method can also include the steps of: calculating anaverage word spacing; selecting an individual word; and repositioningthe individual word within an associated reference or central line;wherein one or more surrounding words are configured to be automaticallyrepositioned along the reference or central line to allow forrepositioning of the individual word in accordance with the average wordspacing on the reference or central line having a proper associatedoffset with respect to the reference or central line.

Similarly, the method can also include the steps of: calculating anaverage character spacing; determine an offset of each character withrespect to an associated reference or central line; selecting anindividual character within an associated word; and repositioning theindividual character within the associated word along the associatedreference or central line; and wherein one or more surroundingcharacters are configured to be automatically repositioned along thereference or central line to allow for repositioning of the individualcharacter in accordance with the average character spacing on theassociated reference or central line having a proper associated offsetwith respect to the reference or central line.

Similarly, the method can also include the steps of: calculating anaverage word spacing; calculating an average character spacing;receiving user input regarding selection of an individual word orcharacter; and receiving a user input correlating to a deletion commandof the word or character within an associated word or line; wherein oneor more surrounding words or characters are repositioned along thereference or central line to allow for repositioning of the individualword or character in accordance with the average word or characterspacing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a single stroke as received by a user inputinterface;

FIGS. 2A-B illustrate screenshot views of various strokes andoverlapping stroke boxes as received by the user input interface;

FIGS. 3A-B illustrates screenshot views of various strokes having bothstroke boxes and stroke margin boxes around each stroke;

FIG. 4 illustrates a screenshot view of various collections ofcharacters being grouped into words, the words having word boxes andword margin boxes surrounding each word;

FIGS. 5A-B illustrate screenshot views of various collections of wordsbeing grouped as a line;

FIG. 6 illustrates a screenshot view of a line box and a calculatedreference or central line through the line box;

FIGS. 7A-B illustrate a line box and reference or central line, whereineach word has a calculated offset from the reference or central line;

FIG. 8 illustrates a screenshot view of a line having a collection ofwords having various user selection and highlighting options;

FIGS. 9A-B illustrate screenshot views of a user selection of a word orcharacter and a relocation option available to the user;

FIGS. 10A-J illustrate screenshot views of a selected word, and amovement transition illustrating an automatic re-spacing of surroundingwords on the line into which the word is being placed as well asre-spacing of surrounding words from the line from which the word isbeing displaced thus illustrating both a respacing during a wordaddition to a line, as well as an associated respacing during a worddeletion;

FIGS. 11A-H illustrate screenshot views of a selected character, and amovement transition illustrating an automatic re-spacing of surroundingcharacters in a word from which the character is being removed ordeleted, additionally the movement transition is illustrated forinserting a character into the word wherein the characters areautomatically spaced apart so as to accommodate the new characteraddition;

FIG. 12 illustrates a screenshot view of a line selection wherein anentire line may be selected and repositioned with regard to adjacentlines;

FIG. 13A-D illustrate screenshot views of various steps of a positiveinference command sequence associating various previously unassociatedstrokes, characters or words;

FIG. 14 illustrates a screenshot view of an object selection andrelocation options of an object or character on the screen;

FIG. 15 illustrates a conceptual diagram of a stroke grouping hierarchy;

FIG. 16 illustrates a front view of an exemplary device having a finalview of a handwritten note as displayed on a user interface of thedevice; and

FIG. 17 illustrates an example schematic of a device or circuitryconfigured to store and perform the various steps and functions of theinvention as contemplated herein.

DETAILED DESCRIPTION

This disclosure relates generally to a system and associated methodswhich are designed to combine some of the benefits of word processingwith the benefits and personalization and stylization of manuallyinputted strokes, i.e. handwriting. As such, methods and systems of thepresent invention will outline a conceptual model and methodology forcombining various core workflows of word processing as we know it withthe much simpler and user-friendly formats of handwritten documents in anew way that works for a wide range of documents, including short notes,letters, as well as more formal and possibly lengthy compositions but iscapable of being utilized across numerous writing styles, languages andcharacter formats.

It will be appreciated that the methods will be discussed as related toknown consumer electronics which include processing circuitry,non-transitory computer-readable media, and user interfaces, for examplepresent tablets or other consumer electronics which can utilize touchscreen or stylus type inputs, alternative uses and platforms capable ofperforming the methods discussed herein will be appreciated by thosehaving skill in the art, such as for three-dimensional design, or invirtual reality workspaces. As such, it will be further appreciated thatthe methods can also be applied to any number of devices that presentlyexist and others that will be developed at a later time.

It will be further appreciated that while the present disclosure refersgenerally to the use of a stylus, virtually all writing implements willbe recognized as a suitable stroke input method, including fingers, handgestures, etc. Additionally, strokes can be recorded in real time, orscanned upon conclusion of a particular input, such as after thecompletion of a sentence. Further, the writing surface can include ascreen, paper, board, either tangible or virtual, or virtually anynumber of surfaces configured to recognize user inputs and, in someinstances, distinguish between various input means particularlyutilizing gestures as input.

The methods of the present invention will involve in-depth discussion ofvarious approaches for breaking down letters, words, and sentences asthey occur in written formats. This is accomplished by treating thestrokes and characters as simple graphic elements and applying generalorganizational formatting rules. This generalization serves to stripaway various complications and inferences that might seem tied to onestyle of handwriting and makes the solution broadly applicable togeneral writing workflows where alternative conventions dominate.Examples of such include right to left rules or top to bottom rules ofwriting, but such rules can be modified or deleted so as to allow forrecognition of characters which occur in various alternative characterlanguages, for example oriental characters, where different rules mightapply. As such, it will be appreciated by those having skill in the art,who also are in possession of this disclosure, that treating letters asgraphic elements makes it possible to accommodate languages that usenon-western alphabets like Chinese, Kanji or Cyrillic. Therefore, whilethe use of the words “letter” or “character” are used throughout thisdisclosure, it will be appreciated that any reference to letters of analphabet are discussing concepts which are equally applicable in manynon-western alphabets, are made up of letters that are more symbolic andconstitute characters or glyphs rather than letters per se.

One underlying concept in handwriting recognition is that each and everystroke 100 of the stylus 50, or other suitable writing utensil, has apath and each path will have a stroke box 110 which is simply a boundingbox that describes the horizontal and vertical extents of the stroke100. An exemplary stroke box 110 is illustrated in FIG. 1. It will beappreciated that while the various paths which ultimately result invarious strokes can be confusing to automatically separate andrecognized, however, it is readily understood that individual strokescan be captured as images or a series of grouped data points rather thanrecognized as alphanumeric characters as they are created, which allowsthem to be captured and processed independently.

It will then be appreciated that various image capturing processes canbe implemented, for example a real-time pointer tracking of a contactpoint or a visual capture between pointer detection and termination ofdetection wherein movement therebetween is tracked and captured.

It will then be appreciated that as stroke data is captured and stored,the path can be simultaneously displayed on a user interface of the userinterface 14, which can be provided in the form of a touch screen and/ordisplay, wherein the user interface is configured so as to essentiallyplay back the gestures that were captured onto a display surface.

It will then be appreciated that the creation of letters or characterscan often require multiple paths/strokes to complete, in which thestylus 50 can be either in constant contact with the surface, or havevarious breaks in contact throughout creation. Additionally, asparticular gestures can be tracked as strokes, which are a grouping ofdistinct but associated data points, the gestures can signal beginningsand endings of strokes in virtual or three-dimensional formats whereinno surface contact is required or even necessary.

As illustrated in FIGS. 2A and 2B we can observe that even when acharacter 200 requires numerous strokes 100, each of these strokes willinclude a stroke box 110 which can contain a beginning point 104 and anend point 108. Additionally, these stroke boxes will most often overlap,thus signifying that while there is a break which creates a “new”stroke, it is nevertheless related to one or more previous strokes, andas such can be correlated together to form a character.

In order to enhance, and adjust, recognition of when multiple pathscreate a single character 200, a stroke margin 120, as illustrated inFIG. 3A, can be provided about the exterior of the stroke box 110 foreach stroke 100. This stroke margin 120 will thus effectively allow fora customizable enlargement of the stroke box so as to allow forenhancement and ultimately increase accuracy of character detection orat least stroke correlations. It will then be understood that the widthsand the heights of the stroke margin 120 can be customized in relationto an associated stroke box 110, for example a wider margin in ahorizontal direction, but less margin in the vertical direction.

Once a character 200 is created, the character can then be provided witha character box 210, as illustrated in FIG. 3B, which reflects thehorizontal and vertical bounds of the strokes forming that character200. In a method similar to that associated with the strokes above, acharacter margin 220 can be generated about the character box 210. Whichcan also be similarly customizable as the stroke margins above, so as tocustomize when or how character margins of adjacent letters overlap,wherein the characters having overlapping character margins can beassociated into words.

Additional information regarding the strokes, such as the thickness, thespeed, the direction, the time and so forth can also be captured inmeta-data. This information can be used to track strokes even when theyoverlap each other.

Once strokes and characters, with their associated margin boxes aregrouped and segregated and stored on a non-transitory computer-readablemedium, the various words, can then be grouped or segregated into lines400, as illustrated in FIG. 4. In order to achieve this correlation,similar to characters and strokes as discussed above, each word 300 canbe provided with an associated word box 310 and a word margin 320,wherein the adjacent word margins or sentences overlap, are correlatedwith one another, and defined or associated with a line 400.

It will also be understood that the stroke, character, or word marginscan be proportional to the overall size of the strokes making up anassociated particular stroke box, character box, or word box. As aproportional value, it allows for a natural adaptation for the size ofstrokes, characters, or words, based on the effective font size which isbeing used and automatically adjusts for the natural increase in spacingused when writing using larger strokes, characters, and words.

It will be understood that one near-universal feature that is common tonearly all handwriting is that people write a series of words or symbolson a baseline. This base line is usually only implied but some writingpaper has printed lines for this purpose. In the method and system ofthe present invention the system 10 can allow for visual graphicpersonalization and/or embellishment of choosing different visualtreatments in the application. In various cases, particularly where theline is not pre-specified on the writing surface, the system can detectan implied line by examining all the bounding boxes of all the words asillustrated in FIGS. 4 and 5A-B. By examining all of the word marginboxes, the system can derive or otherwise determine a line box 410 thatfully encompasses each of the associated word boxes in the same way, byexpressing the extents of all the word boxes in an enclosing box. Thisprinciple is illustrated by FIGS. 5A-B. This line box 510 can oftencontain an entire sentence, but may just be one line of a multi-linesentence.

In some embodiments, the line box can be provided with a line margin,which can then be used to group multiple lines together, which could bebeneficial for forming paragraphs or when writing in multiple columns.As such, while not explicitly illustrated, these margins can also beadded so as to recognize multiple lines in a paragraph, so as to form aparagraph box, so as to scale for larger documents, and the process canbe further repeated so as to implement page boxes and chapters, etc.,until the entire document is complete.

In the present embodiment, a line box 410 has been created whichencompasses an entire line 400 of writing. While it is true that theboxes and the words will often not line up perfectly, a benefit of thisnatural misalignment provides a look and feel of natural handwritingwhich can be measured and detected so as to provide the ability to edit,while maintaining a natural look. Presently, there is a near universalassumption that misalignment is undesirable and needs to be corrected,however, it will be understood that the present invention is intended tocapture natural handwriting and instead it is the imperfections thatmake writing distinctive personal, and unique, which characteristics areintended to be retained by the systems and methods discussed herein.

It will then be appreciated that once the line box 410 has been createdthat each of the word 300 will have a “relative” position within thatline as calculated with respect to a reference or central line 430passing therethrough, as illustrated in FIG. 6. Based on the referenceor central line 430, an offset 330 can be determined for each word 300,as illustrated in FIGS. 7A-B. It will be appreciated that this referenceor central line 430 is free to constantly change since it is derivedfrom the line box 410, which gets its dimensions from the extents of theword boxes 410 contained thereby. The word offsets 330 can berepresented by corresponding distances, which can be expressed asnumbers, or by proportions. Additionally, each and every word's relativeoffset can be based on a distance from each individual word's associatedreference or central line to the reference or central line of the lineof which if forms part. Alternatively, any particular offset can bedetermined by alternative metrics, such as a distance from a referenceor central line to the bottom or top of each word, an average ofindividual characters thereof, etc.

It will also be understood that along with an offset from a reference orcentral line, that an actual or average word spacing can be determinedfor words on a line, or with respect to adjacent words. This spacing canthen be utilized for word selection and repositioning, or editing, aswill be discussed in greater detail below. It will then be appreciatedthat upon insertion, too many words or characters might then exist on aline, in which case words or characters at the beginning or end of aparticular line can be automatically migrated onto previous orsubsequent lines in accordance with the determined spacings for theirnew associated lines.

It will be further appreciated that a similar concept can be employedfor individual characters within a word, wherein a particular wordincludes a reference or central line, and a character offset isdetermined for each character within the word. Similarly, averagecharacter spacing within a word can be determined, or a particularcharacter spacing with respect to adjacent characters can be determined,which can also be useful for editing or other manipulation, as will bediscussed in more detail below.

It will be appreciated that while a user will normally write as if awriting utensil such as a stylus, finger, or other input mechanism werea pencil or otherwise use the input as a typical writing utensil, or andthe user interface were a piece of paper, and that in order to deletetext the only change a simple capture program need do is to undo thewriting of a word or two. However, once a large amount of writing hasbeen captured, the system needs a bit more than a simple undo orworkflow function to adequately edit text. In this manner, the system ofthe present invention utilizes various function controls. All otherdocument operations can be performed in an edit mode. This, edit mode isdesigned to allow a user to edit or manipulate various aspects of thestrokes, characters, words, or lines.

In some instances, certain stroke patterns or motions can act asshortcuts for various editing techniques, for example one such shortcutmotion can include drawing a particular gesture through a word or two soas to delete them, such that the edit mode need not be entered toperform simple tasks. Such an example might include a backward drawn orvertically drawn line through the word. In some such instances, a cursorcan be configured to move to the point of deletion, or alternatively amere highlighting of the word can be presented to the user, such thatthe word can simply be re-written, wherein the words previously writtenafter the deleted word can be configured to auto-space so as to allowfor the new word to be inserted before them.

In some instances, the user interface can be configured to detectdifferences between various input means, such as between a stylus input,which can be configured as a positive input means, versus a finger ortouch input which can be configured as a negative input means, i.e. adeletion. For example, a person can be positively writing with thestylus and the associated strokes showing on the screen with eachstroke, character, word, and line associations, but a swipe of thefinger can be utilized for an alternative command, such as to undo animmediately previous stroke. Alternatively, a touch input can beutilized as a digital eraser which can be utilized anywhere on the userinterface to delete strokes, characters, words, lines, etc. which werepreviously provided.

By default, the system can automatically enter the edit word mode, butthe system can also be configured to toggle between edit word and editline modes or toggle between selection modes wherein individual strokes,characters, words, or lines can be selected based on varying categoriesof correlation. It will be readily apparent when the system is in editmode as the words the user wrote will have visible word or margin boxesand can allow for making selections for editing, manipulating,repositioning, rotating, or deleting those selections. The mechanics ofselection chains will be covered in more detail below.

Edit line mode can allow you to make global adjustments to a handwrittenline of text wherein the user can adjust the line in various directions.In particular, in edit line mode the number of words or characters whichcan be selected can include the entire line, or a selection multiplewords or characters within that line.

When writing, occasionally there will be the need to undo some gestures;with pencil and paper, trying to erase individual strokes ends up with apoor result most of the time, therefore unlike using a typewriter, thesmallest element you can reasonably undo is a word. While there is agranularity of undo which is proper since rewriting the word is the mostintuitive place to start; however, not only does it take a lot of time,relatively speaking, but it breaks the cognitive model of writing,causing the writer/user to break or change their line of thought.

When a user has one or more words that are embedded in the document suchthat “Undo” is the wrong solution, changing into edit mode can affordadditional options for word deletion, manipulation, or editing. In someinstances, tapping inside a word box can toggle selection, but startinga stroke outside of a word box will enable a selection stroke wherecontinuing the stroke into a word box, will select that word; this staysin effect until the stroke ends by lifting the stylus. This way, itbecomes possible to chain selections of multiple words by continuing thestroke past the first word box into multiple word boxes, effectivelyselecting all the words. This is similar to simply striking through oneor more boxes. Once there are selected words, they can be highlighted asillustrated in FIG. 8, in order to show that they are selected. Theselected words can be deleted with a single gesture, such as via awastebasket icon button, or a particular gesture or stroke while in theedit mode.

In edit word mode words can be selected and deselected; one way to doindicate word selection mode is to make the word boxes visible to theuser as illustrated in FIG. 9B. From there, a predetermined user gesturecan toggle selection of particular word boxes on and off, word by wordand various modes and manipulation menus can be accessed so as toachieve the desired edit based on the input user gesture.

In this manner, the words in a document can be moved, such as via a dragoperation to another position in the document. FIGS. 10A-J illustratevarious operations which occur upon selection and a relocation of aparticular word. It will be appreciated that upon insertion of a wordonto a particular line, that the surrounding words can be configured tomove or translate along the line so as to make room for the insertion.Additionally, in a drag operation, which is essentially a deletion of aword from a line, the words can be configured to translate into thepreviously occupied space so as to close up the space where the removedword used to be.

In some instances, the average spacing between adjacent words can bedetermined as the words of the line are recognized, this word spacingcan be determined, and a relocation of the word can cause all the wordson the line to automatically reposition in accordance with thedetermined word spacing such that the flow of the line appears naturalwith respect to the other lines or the previously detected spacing. Thiscalculated spacing can be used for both insertion, as shown in FIGS.10F-J, onto a line as well as for a word deletion, as illustrated inFIGS. 10A-E.

In some such embodiments, if there is insufficient room on the line toaccommodate the insertion of a particular ward such that there isinsufficient room on the line for each of the pre-existing wordsthereon, the end of the line can then be truncated and wrapped onto tothe next line.

Similar mechanics can be utilized for the change, addition, or deletionof a character within a word, as illustrated in FIGS. 11A-H. Wherein acharacter spacing within a particular word is determined, and upondeletion, such as illustrated in FIGS. 11C-E, the characters areautomatically repositioned to the natural spacing, or upon addition,such as illustrated in FIGS. 11F-H, wherein the adjacent characters arerepositioned to allow for insertion of the character in a mannerreflective of the determined natural spacing between characters.

FIG. 12 also illustrates how a particular line 400 can be selected andrepositioned with respect to adjacent lines so as to change the relativespacing of the entire line, so as to line up intended paragraphs, reducethe height of a paragraph so as to allow for insertion of another line,such as when running out of space on the bottom or edge of a page, etc.

In yet other aspects of the present invention, it has been recognizedthat variations in spacing often occur between strokes, characters, orwords. Such an instance is illustrated in FIGS. 13A-D. Illustrated hereis an instance in which the user has written various characters whichare intended to be in a single word, but because of variation in thespacing, which can be caused due to numerous factors, such as handrepositioning while writing, or just through common mistake, the endcharacters are spaced from the previous characters by too wide a margin,and as such there is no overlap between the character boxes or thecharacter margins and as such are not associated together.

It has then been recognized that a teaching sequence can be performed inwhich the user can provide what will be referred to herein as aninference stroke 150, spanning the gap between the unintentionallyspaced characters. The inference stroke will then be recognized as astroke connecting them, and can include a stroke margin 154 surroundingthe inference mark, and a stroke margin as well. The inference stroke150 can then be deleted, but the association between the charactersretained, so as to retain the grouping of characters into a common word.

In some such instances, the processing circuitry 18 can be configured todifferentiate between a plurality of input methods including at least atactile touch input and input received by the writing utensil, such asthe stylus 50, wherein input from the writing utensil is associated witha positive stroke, and touch input is associated with an alternativecommand. In other words, the inference stroke 150 can be provided usinga stylus, however the deletion of the inference stroke 150, i.e.alternative command, can be achieved by providing a gesture using analternative means, such as a finger swipe provided by the user's finger54.

In some such embodiments, the alternative command can be an undo commandregarding at least one previous stroke, wherein an inference strokemargin 154 of the inference stroke 150 can be retained, even if thestroke itself is deleted.

As such, the processing circuitry can be configured to receive from thewriting utensil, a positive inference stroke connecting a plurality ofseparated strokes; and receive a touch input so as to delete thepositive inference stroke, wherein the processing circuitry causes thenon-transitory computer-readable media to retain inference informationbetween the strokes so as to recognize the plurality of separatedstrokes as an associated character, word, or line. so as to achieve thedesired result as discussed above.

This inference sequence, as discussed above, can be automaticallydetected, only active in edit mode, or the particular functions of thissequence can be toggled on or off in different editing modes, etc.

In some embodiments, this inference sequence can also be configured toautomatically adjust the margin distances for strokes, characters, orwords, depending on which types of groups are being connected by aparticular inference stroke so as to allow for a degree of automaticlearning and adaptation of the program to a particular user's naturalstroke, character, or word spacing.

Additionally, sketches, or other quick hand-drawn sketches can also beconfigured to be incorporated into the system. It is beneficial torecognize that a sketch can be handled as a more complex letter which issimply made up of many more strokes. In some instances, a sketchrecognition can merely be a different mode which is automaticallyentered as the processing circuitry recognizes a threshold value ofassociated or overlapping strokes. Sketches will also have an ultimatesize that crosses multiple lines, whereas, normal letters of an alphabetfit inside the bounding box as expected. Sketches are therefore aspecial case that necessitates an alternative method of processing.

In certain embodiments, and as illustrated in FIG. 14, sketches 500 canbe configured to be recognized as a floating graphic that can be movedaround in an edit mode and the text would flow around the sketch. Itwill be understood that moving a sketch up in this case into existingtext, that the text can be repositioned to allow for the insertion ofthe sketch.

It will then be appreciated that because in the system described abovethere is no pre-set character or language, and that the characters,letters, and lines are just a collection of one or more overlappingstrokes. As such, one difficulty which arises is that there is alwaysthe chance that free-floating punctuation, accents, and/or grammaticalmarks which may either be too far from a word to be considered properlyassociated with word, or in some instances the punctuation can beintentionally unassociated with the work. In some such instances, theassociation between characters and punctuation can require apre-determined size ratio between a particular stroke and a pre-existingcharacter, i.e. a group of associated strokes, before an association iscreated and the stroke incorporated into the character. In order tooverride the ration requirement, an inference mark and deletion can beutilized to force an association. For example, a period at the end of asentence is small with respect to the adjacent word, and no inferencethat the period is part of the word is desired. As such, a size rationthreshold can be implemented which would exclude the inclusion of theperiod. Alternatively, dotting an “i” would also not be automaticallyassociated with the below mark in such an instance, but an inferencestroke could be created to connect the two strokes and subsequentlydeleted while maintaining the inference should the user subsequentlydesire to reposition the word or character.

Shown below in FIG. 16 is an exemplary device 12 illustrating a screenshot of the writer screen on the user interface 14 based on the variousembodiments and concepts discussed in this document. It will beappreciated that in some instances, or in particular modes, the userinterface 14 can be configured to intentionally omit many typicalcomplex interface options such as toolbars, formatting options, etc., aseach of the commands are performed using distinct strokes and stylusbehaviors.

FIG. 15 illustrates a conceptual diagram of a stroke grouping hierarchy30 which illustrates the segregation of strokes 100, characters 200,words 300, and lines 400, vs. strokes 100 which are ultimatelyrecognized as sketches 500, which can then be incorporated ontooverarching pages 600, and entire documents 700.

Additionally, in some embodiments, the input strokes can be translatedinto a computer Unicode character set so as to allow for the adaptationof the methods in the recognition of various language characters andwriting formats, which allow for right-to-left and/or top to bottomwritten languages.

It will also be appreciated that since the characters themselves are notbeing translated into a particular code or font, but are rather treatedas objects which are grouped together using a stroke proximity margin,that it will be exceedingly difficult to process these documents withoptical recognition methods typically employed by standard computers. Assuch, when combined with cryptography, new written languages or codescan be created that would be theoretically unbreakable or otherwiseincreasingly secure.

In particular, there are simple encoding schemes that are extremelydifficult to break without a specific key. The present invention has theadvantage of not requiring cryptography, which is often required tosecure things written in plain English as the strokes are a grouping ofindividual data points rather than recognized characters.

FIG. 17 illustrates an example schematic of a device or circuitry 10configured to store and perform the various steps and functions of theinvention as contemplated herein. The device 10 can be equipped with auser interface 14 for receiving inputs, such as strokes, processingcircuitry 18 can analyze those inputs, such as the strokes and determinevarious bounding boxes and relationships associated with the strokes asdescribed above, and non-transitory computer readable media 22 can storeinformation association with those inputs, as well as instructionalinformation from which the processing circuitry can follow commands orrules to perform the various calculations and determinations discussedabove.

Various embodiments and options have been described herein that areexemplary of the present invention; one skilled in the art willrecognize additional embodiments within the spirit and scope of theinvention. Modification and variations can be made to the disclosedembodiments without departing from the scope of the disclosure. Thoseskilled in the art will appreciate that the applications of theembodiments disclosed herein are varied. Accordingly, additions andmodifications can be made without departing from the principles of thedisclosure. In this regard, it is intended that such changes would stillfall within the scope of the disclosure. Therefore, this disclosure isnot limited to the particular embodiment as shown, but is intended tocover modifications within the spirit and scope of the disclosure.

What is claimed is:
 1. A handwriting recognition and processing system,the system comprising: a non-transitory computer-readable medium; aninput recognition interface; and processing circuitry operably connectedto the non-transitory computer-readable medium and the input recognitioninterface, the processing circuitry being configured to perform thefollowing tasks: recognize a plurality of strokes provided on the inputrecognition interface; determine a stroke box around each stroke;provide a stroke margin around each of the stroke boxes; determineoverlap between the stroke margins for each stroke box; correlateoverlapping stroke margins to one or more characters; provide acharacter box around each of the one or more characters; and provide acharacter margin around each of the one or more character boxes.
 2. Thehandwriting recognition and processing system of claim 1, wherein theprocessing circuitry is further configured to perform the followingtasks: determine overlap between the character margins for eachcharacter box; and correlate overlapping character margins to one ormore words.
 3. The handwriting recognition and processing system ofclaim 2, wherein the processing circuitry is also configured todifferentiate between a plurality of input methods including at least astylus input and a touch input.
 4. The handwriting recognition andprocessing system of claim 3, wherein input from the stylus isassociated with a positive stroke, and touch input is associated with analternative command.
 5. The handwriting recognition and processingsystem of claim 4, wherein at least one alternative command is an undocommand regarding at least one previous stroke.
 6. The handwritingrecognition and processing system of claim 5, wherein the inputrecognition interface can be configured to receive a positive inferencestroke connecting a plurality of separated strokes from the stylus, anda touch input received to delete the positive inference stroke, whereinthe processing circuitry retains inference information between thestrokes so as to recognize the plurality of separated strokes as anassociated character, word, or line.
 7. The handwriting recognition andprocessing system of claim 2, wherein the processing circuitry isfurther configured to perform the following tasks: provide a word boxaround each of the one or more words; provide a word margin around eachof the one or more word boxes; determine overlap between the wordmargins for each word box; correlate overlapping word margins to one ormore lines; and determine a line box correlating to the vertical andhorizontal bounds of each line.
 8. The handwriting recognition andprocessing system of claim 7, wherein the processing circuitry isfurther configured to perform the following tasks: determine a referenceline for each line box; and determine an offset of each word withrespect to the reference line.
 9. The handwriting recognition andprocessing system of claim 8, wherein the processing circuitry isfurther configured to perform the following tasks: calculating anaverage word spacing; allow selection of an individual word; and allowfor repositioning of the individual word within an associated referenceline; and wherein one or more surrounding words are configured to beautomatically repositioned along the reference line to allow forrepositioning of the individual word in accordance with the average wordspacing on the reference line having a proper associated offset withrespect to the reference line.
 10. The handwriting recognition andprocessing system of claim 8, wherein the processing circuitry isfurther configured to perform the following tasks: calculating anaverage character spacing; determine an offset of each character withrespect to an associated reference line. allow selection of anindividual character within an associated word; and allow forrepositioning of the individual character within the associated wordalong the associated reference line; and wherein one or more surroundingcharacters are configured to be automatically repositioned along thereference line to allow for repositioning of the individual character inaccordance with the average character spacing on the associatedreference line having a proper associated offset with respect to thereference line.
 11. The handwriting recognition and processing system ofclaim 2, wherein the processing circuitry is further configured toperform the following tasks: calculating an average word spacing;calculating an average character spacing; receiving user input regardingselection of an individual word or character; and receiving a user inputcorrelating to a deletion command of the word or character within anassociated word or line; wherein one or more surrounding words orcharacters are repositioned along the reference line to allow forrepositioning of the individual word or character in accordance with theaverage word or character spacing.
 12. A method of recognizing andprocessing hand-written strokes and characters, the method comprising:receiving a plurality of strokes provided by using a writing utensil onan input recognition interface; recognizing the plurality of strokesprovided on the input recognition interface utilizing processingcircuitry; determining a stroke box around each stroke; providing astroke margin around each of the stroke boxes; determining an overlapbetween the stroke margins for each stroke box; correlating anyoverlapping stroke margins to one or more characters; providing acharacter box around each of the one or more characters; providing acharacter margin around each of the one or more character boxes;determining overlap between the character margins for each characterbox; correlating overlapping character margins to one or more words; andstoring stroke and character information on a non-transitorycomputer-readable medium.
 13. The method of recognizing and processinghand-written strokes and characters of claim 12, further comprising:differentiating between a plurality of input methods including at leasta tactile touch input and input received by the writing utensil, whereininput from the writing utensil is associated with a positive stroke, andtouch input is associated with an alternative command.
 14. Thehandwriting recognition and processing system of claim 13, wherein atleast one alternative command is an undo command regarding at least oneprevious stroke.
 15. The handwriting recognition and processing systemof claim 14, further comprising: receiving, from the writing utensil, apositive inference stroke connecting a plurality of separated strokes;and receiving a touch input so as to delete the positive inferencestroke, wherein the processing circuitry causes the non-transitorycomputer-readable media to retain inference information between thestrokes so as to recognize the plurality of separated strokes as anassociated character, word, or line.
 16. The method of recognizing andprocessing hand-written strokes and characters of claim 12, furthercomprising: providing a word box around each of the one or more words;providing a word margin around each of the one or more word boxes;determining an overlap between the word margins for each word box;correlating overlapping word margins to one or more lines; determining aline box correlating to the vertical and horizontal bounds of each line;determining a reference line for each line box; and determining anoffset of each word with respect to the reference line.
 17. The methodof recognizing and processing hand-written strokes and characters ofclaim 16, further comprising: calculating an average word spacing;selecting an individual word; and repositioning the individual wordwithin an associated reference line; and wherein one or more surroundingwords are configured to be automatically repositioned along thereference line to allow for repositioning of the individual word inaccordance with the average word spacing on the reference line having aproper associated offset with respect to the reference line.
 18. Themethod of recognizing and processing hand-written strokes and charactersof claim 16, further comprising: calculating an average characterspacing; determine an offset of each character with respect to anassociated reference line. selecting an individual character within anassociated word; and repositioning the individual character within theassociated word along the associated reference line; and wherein one ormore surrounding characters are configured to be automaticallyrepositioned along the reference line to allow for repositioning of theindividual character in accordance with the average character spacing onthe associated reference line having a proper associated offset withrespect to the reference line.
 19. The method of recognizing andprocessing hand-written strokes and characters of claim 16, furthercomprising: calculating an average word spacing; calculating an averagecharacter spacing; receiving user input regarding selection of anindividual word or character; and receiving a user input correlating toa deletion command of the word or character within an associated word orline; wherein one or more surrounding words or characters arerepositioned along the reference line to allow for repositioning of theindividual word or character in accordance with the average word orcharacter spacing.